Registration of separations

ABSTRACT

Separations or images relating to film or other fields may be registered using a variety of features, such as, for example: (1) correcting one or more film distortions; (2) automatically determining a transformation to reduce a film distortion; (3) applying multiple criteria of merit to a set of features to determine a set of features to use in determining a transformation; (4) determining transformations for areas in an image or a separation in a radial order; (5) comparing areas in images or separations by weighting feature pixels differently than non-feature pixels; (6) determining distortion values for transformations by applying a partial distortion measure and/or using a spiral search configuration; (7) determining transformations by using different sets of features to determine corresponding transformation parameters in an iterative manner; and (8) applying a feathering technique to neighboring areas within an image or separation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/347,485, filed Jan. 10, 2012, now allowed, which is a continuation ofU.S. patent application Ser. No. 12/712,888, filed Feb. 25, 2010, nowU.S. Pat. No. 8,094,972, which is a continuation of U.S. patentapplication Ser. No. 11/929,191, filed Oct. 30, 2007, now U.S. Pat. No.7,672,541, which is a divisional of U.S. patent application Ser. No.11/856,541, filed Sep. 17, 2007, now U.S. Pat. No. 7,486,842, which is acontinuation of U.S. patent application Ser. No. 11/282,675, filed Nov.21, 2005, now U.S. Pat. No. 7,272,268, which is a divisional of U.S.patent application Ser. No. 10/035,337, filed Jan. 4, 2002, now U.S.Pat. No. 7,092,584, the entire contents of the prior applications beingincorporated herein by reference.

TECHNICAL FIELD

This invention relates to image processing, and more particularly to theregistration of separations.

BACKGROUND

Color motion picture film is a relatively recent development. Before theadvent of color film stock in the 1950s, a process for making colormotion pictures included capturing color information on two or morereels of black and white film. In the original Technicolor three colorfilm separation process, three reels of black and white film were loadedinto a specially-designed movie camera. The light coming through thelens was split into the three primary colors of light and each wasrecorded on a separate reel of black and white film. After developingthe three reels, three photographic negatives representing the yellow(inverted blue), the cyan (inverted red), and the magenta (invertedgreen) portion of the original reels were created.

In addition to the creation of color separations through the originalTechnicolor process, color separations also have been produced and usedfor the archival of color film because black and white film stockgenerally has a much greater shelf-life than color film. In thisprocess, the color film stock is used to expose one reel of black andwhite film with sequential records of red, green, and blue so that eachframe is printed three times on the resultant reel to form a sequentialseparation.

Film studios may recombine the three color separations onto a singlereel of color film using a photographic process that is performed in afilm laboratory. In the case of three color separations that are eachlocated on a separate reel, an optical film printer is employed toresize and reposition each source reel, one at a time. In particular,three passes are made. First, the magenta source reel is projectedthrough an appropriate color filter onto the destination reel.Thereafter, the destination reel is rewound, the next source reel isloaded and resized, and the color filter is changed. For this reel, ahuman operator determines a global alignment (and scaling if necessary)for the entire set of frames within the reel or, alternatively, withinselected scenes on a scene-by-scene basis, with each scene includingseveral, if not hundreds, of frames. However, because of the humanintervention required, the alignment often is not determined on aframe-by-frame basis for the entire reel. The process is repeated untilall three color separations have been printed on the single destinationreel using the optical film printer. The resulting destination reel iscalled an interpositive (“IP”), and the colors are now represented asred, green, and blue (as opposed to cyan, magenta, and yellow).

SUMMARY

In one general aspect, automatic registration of film separationsincludes accessing component images that are based on digitized filmseparations. Each of the component images includes a set of gray-levelpixels. An alignment vector is automatically determined for at least apart of a selected component image from among the accessed componentimages. One or more film distortions are corrected by applying thealignment vector to the part of the selected component image.

Component images may be based on digitized color film separations.Accessing component images may include digitizing film separations.

The alignment vector may align a part of a component image with acorresponding part of another component image, and the two componentimages may be combined after applying the alignment vector. Onecomponent image (e.g., a green component image) may be selected as areference, and an alignment vector may be determined between a part ofthe reference and a part of another component image. An additionalalignment vector may be determined between a part of a reference and apart of an additional component image.

An alignment vector may be determined by determining a first set offeatures associated with a part of a reference and determining a secondset of features associated with a part of a selected component image.The first and second sets of features may be compared based on resultsobtained when applying one or more candidate alignment vectors. Thealignment vector may be determined based on results of the one or morecomparisons.

A first set of features may be determined by applying an edge detectionfilter to a part of a reference to generate a first preliminary set ofedges. An edge refinement procedure may be applied to the firstpreliminary set of edges to obtain the first set of features. The secondset of features may be determined by applying the edge detection filterto a part of a selected component image to generate a second preliminaryset of edges. The edge refinement procedure may be applied to the secondpreliminary set of edges to obtain the second set of features. The edgerefinement procedure may include selecting edges based on acharacteristic (e.g., high intensity) of a component image.

The edge refinement procedure may include identifying a connected edgewithin an area under consideration. The connected edge may be includedin a set of selected edges if the connected edge meets a first criterionof merit. The first criterion of merit may require at least apredetermined amount of information in one direction. A set of featuresmay be obtained based on whether the entire set of selected edgessatisfies a second criterion of merit. The second criterion of merit mayrequire the entire set of selected edges to have at least apredetermined amount of information.

Comparing a first and a second set of features may include assigning anon-zero amount of distortion to a pixel in a first component image onlyif the pixel is part of a feature and if a pixel at a correspondinglocation in a second component image is not part of a feature. Thedistortion values obtained for a predefined set of pixels in an areabeing examined in the first component image may be summed. The firstcomponent image may be any component image.

Comparing a first and second set of features based on results obtainedwhen applying a candidate alignment vector may include selecting aninitial candidate alignment vector. The initial candidate alignmentvector may be varied so as to represent multiple relative displacementpossibilities within a particular proximity window of the initialcandidate alignment vector. Selecting the initial candidate alignmentvector may include determining a first set of features associated with acenter part of a reference, and determining a second set of featuresassociated with a center part of a selected component image. The firstand second sets of features associated with the center parts of thereference and the selected component image may be compared, and theinitial candidate alignment vector may be selected based on results ofthe comparison.

Determining an alignment vector may include dividing a selectedcomponent image into a set of areas. An initial alignment vector for aparticular area may be determined based on at least one previouslydetermined alignment vector for another area. The alignment vector forthe particular area may be determined based on the initial alignmentvector for the particular area, which may be determined based on atleast one previously determined alignment vector for a neighboring area,which may be defined as an area that shares a common border or at leastone pixel with the particular area. The initial alignment vector may bechosen as the previously determined alignment vector that provides aminimum distortion value for the particular area among the previouslydetermined alignment vectors for at least two of the neighboring areas.

Alignment vectors may be applied to multiple areas of a component image,and a technique may be applied to smooth discontinuities that may resultwhen different areas possess different alignment vectors. Applying atechnique to smooth discontinuities may include defining a window ofnonzero horizontal or vertical extent along a boundary of contiguousblocks. Alignment vectors obtained from each of the contiguous blocksmay be interpolated in order to obtain a new set of alignment vectorsfor parts of the contiguous blocks within the window. The new set ofalignment vectors may be applied to the parts of the contiguous blockswithin the window.

Determining an alignment vector may include dividing the selectedcomponent image into a set of areas arranged such that a center of atleast one area of the set of areas and a center of at least one otherarea of the set of areas are in different proximity to a center of theselected component image. The areas are grouped into multiple rings. Aninitial alignment vector for a particular area is determined based on atleast one previously determined alignment vector for at least oneneighboring area, where the neighboring area belongs to either an innerring or to the same ring as the particular area. The alignment vectorfor the particular area is determined based on the initial alignmentvector for the particular area. The initial alignment vector may bedetermined based on a previously determined alignment vector thatprovides a minimum distortion value for the particular area amongpreviously determined alignment vectors for at least two neighboringareas. The initial alignment vector may be determined based on apreviously determined alignment vector for an inward radial neighborarea.

In another general aspect, performing registration of digitized imagesincludes selecting at least two areas from each of a first image and asecond image. Separate transformations are determined for the selectedareas of the first image based on a comparison of areas within the firstand second images. A feathering technique is applied within apredetermined amount of at least two neighboring areas within theselected areas in order to obtain new transformations for thepredetermined areas if the transformations for the neighboring areasdiffer, where the new transformations are based on the separatetransformations.

The transformations may be applied to the selected areas of the firstimage. The transformations may be represented as alignment vectors, andapplying the feathering technique may include linearly interpolatingbetween alignment vectors. The images may correspond to color filmseparations.

In another general aspect, performing registration of digitized imagesincludes dividing a selected component image into a set of areas andgrouping the areas into multiple rings. Transformations for at least twoareas are determined in an order that begins with at least one areawithin an innermost ring and proceeds to at least one area within a ringother than the innermost ring.

The set of areas may be arranged such that a center of at least one areaof the set of areas and a center of at least one other area of the setof areas are in different proximity to a center of the selectedcomponent image. Determining the transformations may include determiningan initial alignment vector for a particular area of the set of areasbased on a previously determined alignment vector corresponding to atleast one neighboring area, where the neighboring area belongs to eitheran inner ring or to a same ring as the particular area. An alignmentvector may be determined for the particular area based on the initialalignment vector for the particular area. The initial alignment vectormay be based on a previously determined alignment vector that provides aminimum distortion measure for the particular area among previouslydetermined alignment vectors for at least two neighboring areas. Theinitial alignment vector may be based on a previously determinedalignment vector for an inward radial neighboring area.

In another general aspect, performing registration of digitized imagesincludes selecting a first area in each of a first image and a secondimage. Feature pixels are determined in the first areas of the first andsecond images. The first areas are compared by weighting a comparison offeature pixels in the first area of the first image with correspondingpixels in the first area of the second image differently than acomparison of non-feature pixels in the first area of the first imagewith corresponding pixels in the first area of the second image. Atransformation is determined for the first area of the first image basedon the comparison of the first areas.

The first and second images may be based on digitized color filmseparations. Weighting may include accumulating a non-zero distortiononly if a pixel in the first area of the first image has been classifiedas a feature and a corresponding pixel in the first area of the secondimage has not been classified as a feature. The features may be edges.Weighting may include associating a weight of zero to the comparison ofnon-feature pixels in the first area of the first image withcorresponding pixels in the first area of the second image, such thatthe comparison involving non-feature pixels in the first area of thefirst image need not be performed.

In another general aspect, performing registration of digitized imagesincludes selecting a first area in each of a first image and a secondimage. Feature pixels are determined in the first areas of the first andsecond images. A transformation is determined for the first area of thefirst image. Determining the transformation includes computingdistortion values using a partial distortion measure on candidatealignment vectors that are processed in a spiral search configuration,and selecting one of the candidate alignment vectors as thetransformation based on the computed distortion values.

The spiral search may include determining distortion values associatedwith different horizontal and vertical relative displacements of aninitial alignment vector in an order characterized by increasing radialdistance along a spiral scanning path. Determining distortion valuesassociated with different horizontal and vertical relative displacementsmay include beginning at a location associated with the initialalignment vector and proceeding along the spiral scanning path within apreset window size.

Computing distortion values using the partial distortion measure mayinclude defining a set of pixels within an area. A partial sum ofdistortion values may be calculated, the partial sum being associatedwith a candidate alignment vector using a subset of the set of pixels.The partial sum may be compared to a current minimum distortion. Thecandidate alignment vector may be excluded as a potential choice for thetransformation if the partial sum is greater than or equal to thecurrent minimum distortion. An additional partial sum may be added tothe partial sum if the partial sum is less than the current minimumdistortion, the additional partial sum being obtained using anadditional subset of the set of pixels. Further additional partial sumsmay be added, and further comparisons to the current minimum distortionmay be performed, until either the partial sum is greater than or equalto the current minimum distortion or all pixels in the set have beenused.

In another general aspect, performing registration of digitized imagesincludes selecting a first image and a second image. A first set offeatures and a second set of features are defined. A first alignmentvector is determined for a part of the first image based on the firstset of features. A second alignment vector is determined for the part ofthe first image based on the second set of features. Determining thesecond alignment vector includes using the first alignment vector as aninitial second alignment vector, and choosing the second alignmentvector from a set of candidate alignment vectors obtained by varying theinitial second alignment vector. The first alignment vector is modified.Modifying the first alignment includes using the second alignment vectoras an initial first alignment vector, and choosing the first alignmentvector from a set of candidate alignment vectors obtained by varying theinitial first alignment vector. Determining the second alignment vectorand modifying the first alignment vector are repeated until a particularstopping condition is met.

The first set of features may correspond to edges in one direction andthe second set of features may correspond to edges in an orthogonaldirection. The set of candidate alignment vectors for each directionalset of edges may include alignment values that differ in only onedirection. The set of candidate alignment vectors may decrease in sizeeach time the first and second alignment vectors are determined. Thestopping condition may be a preset number of iterations. The stoppingcondition may be met when the first and second alignment vectorsdetermined after a particular iteration are equivalent to the first andsecond alignment vectors after a previous iteration.

In another general aspect, performing registration of digitized imagesincludes selecting a first area from each of a first image and a secondimage. A first set of features is detected in the first area of thefirst image. A second set of features is determined consisting of thefeatures from within the first set that include a first predeterminedamount of information. It is determined whether the second set offeatures is collectively sufficient to provide a meaningful comparison.If the second set of features is deemed collectively sufficient, atransformation is determined for the first area of the first image basedon a comparison of the first area of the first image with areas of thesecond image.

The second set of features may be sufficient if the second setcollectively includes a second predetermined amount of information. Thefirst set of features may include a set of features that are oriented ina particular direction. The first set of features may include edges inthe first area of the first image. The images may be based on digitizedspectral separations. The spectral separations may include color filmseparations.

The first areas of the first and second images may include pixels.Determining a transformation may include determining which pixels in thefirst areas of the first and second images are feature pixels andweighting them differently. Weighting may include weighting a comparisonof feature pixels in the first area of the first image withcorresponding pixels in the first area of the second image differentlythan a comparison of non-feature pixels in the first area of the firstimage with corresponding pixels in the first area of the second image.

In another general aspect, performing registration of digitized imagesincludes selecting automatically a first area in each of a first imageand a second image. A transformation is determined automatically for thefirst area in the first image based on a comparison of the first area ofthe first image with corresponding areas of the second image. Thetransformation is applied automatically to the first area in the firstimage, and a film distortion is reduced.

The first and second images may be based on film separations, and thefilm separations may include color separations. A transformation may besimilarly determined and applied to a second area in the first imagethat is not isolated from the first area. The two transformations may berepresented as alignment vectors that differ and a feathering may beapplied. A feathering technique may be applied to the first and secondareas within a predetermined amount of the first and second areas inorder to obtain new alignment vectors within the predetermined amounts,where the new alignment vectors are based on the first and secondalignment vectors. Transformations may be similarly determined andapplied to multiple areas in the first image, and the transformationsfor the multiple areas may be determined in an order of increasingradial distance. Determining the transformation may include selectingautomatically a feature in the first areas of the first and secondimages and using a feature-based measure to compare the first areas ofthe first and second images.

Determining the transformation may include detecting automatically afeature in the first area of the first image. Parts of the feature maybe eliminated automatically if they do not contain a predeterminedamount of information. It may be determined whether parts of the featurethat were not eliminated provide a basis for meaningful comparison.Determining whether the parts not eliminated provide a basis formeaningful comparison may include determining whether the parts noteliminated collectively contain a second predetermined amount ofinformation. The feature may include edges in the first area.

Other features and advantages will be apparent from the followingdescription, including the drawings, and the claims.

Various general aspects address methods for the registration ofseparations or images that may correspond to separations. The separationmay relate to film or other fields. Such methods may include one or moreof a variety of features, such as, for example: (1) accessing componentimages that are based on a variety of separations, including, forexample, digitized film separations for which each of the componentimages includes a set of gray-level pixels; (2) correcting one or morefilm distortions; (3) automatically determining a transformation toreduce a film distortion; (4) applying multiple criteria of merit to aset of features to determine a set of features to use in determining atransformation; (5) determining transformations for areas in an image ora separation in a radial order; (6) comparing areas in images orseparations by weighting feature pixels differently than non-featurepixels; (7) determining distortion values for transformations byapplying a partial distortion measure and/or using a spiral searchconfiguration; (8) determining transformations by using different setsof features to determine corresponding transformation parameters in aniterative manner; and (9) applying a feathering technique to neighboringareas within an image or separation.

The described implementations may achieve, one or more of the followingfeatures. For example, they may provide an automatic and efficientdigital image registration process for color film separations. Theprocess may operate in the digital domain to enable the use of a numberof digital image processing techniques, and may require minimal humanintervention. The process may be computationally efficient, and may becapable of determining alignments on a frame-by-frame basis. The processalso may address the local nature of the misregistration within an imagethat results from such causes as film shrinkage due to aging. Inaddition, the process may compensate, correct, or avoid one or more ofthe described distortions.

One or more implementations are set forth in the accompanying drawingsand the description below. Other implementations will be apparent fromthe description, drawings, and claims.

DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or application publication with colordrawing(s) will be provided by the Office upon request and payment ofthe necessary fee.

FIG. 1 is a picture illustrating misregistration.

FIG. 2 is a block diagram of an implementation of a registration method.

FIG. 3 is a diagram of a partitioning of an image into blocks.

FIG. 4 is a diagram of one implementation of a processing order for thepartition of FIG. 3.

FIG. 5 is a diagram highlighting areas in which one implementationapplies feathering.

FIG. 6 is a picture illustrating a result of applying one implementationof a registration technique to the picture in FIG. 1.

DETAILED DESCRIPTION

The film processes described earlier, as well as other processes, may besubject to one or more of a variety of well-known film distortions.These include static misregistration, dynamic misregistration,differential resolution, and loss of resolution. Although referred to asfilm distortions, these distortions also may be present in otherapplications and environments. For example, registration of separationsmay be required, and one or more film distortions may be present, inphotography, astronomy, and medical applications.

Static misregistration may be experienced due to one or more of avariety of reasons, six examples of which follow. First, in order toprevent color fringing around objects, the three source separationsshould be aligned with each another. The original cameras typically wereadjusted mechanically by a technician with a micrometer. The alignmentresults therefore often varied, usually from camera to camera, within asingle movie title. Second, due to variations among film printers, theprinters sometimes failed to hold each of the separations to the sametolerances. Third, differences between the camera and the printer mayhave caused color shifting. Fourth, the photographic compositing processdiscussed above typically operates on either a reel-by-reel basis or ascene-by-scene basis, rendering it difficult to correct misalignmentsthat may occur on a frame-by-frame basis. Fifth, because thephotographic compositing process provides a global alignment for aparticular image, the process does not necessarily address the localmisalignments that may occur within an image. Sixth, because film tendsto shrink as it ages and the rate of shrinkage may vary per separation,color can fringe around the edges of an image even if the center of theimage is perfectly registered.

Dynamic misregistration also may be experienced due to one or more of avariety of reasons, three examples of which follow. First, the filmseparations in a camera are subject to intermittent motion, stopping andstarting, for example, twenty four times every second. All threeseparations must stop in precise alignment in order to obtain properregistration. However, such precise timing is difficult to achieve andmaintain. Second, from frame-to-frame, the film may move in the cameraor subsequent film printer leading to color fringing that moves in likemanner. Third, film may be spliced together as part of a normal editingprocess, resulting in splices that are physically thicker than the film.When the splices pass over a roller in the photographic printingprocess, a small vertical bump in one or more color film separations mayoccur. As discussed above, because the photographic compositing processmay not operate on a frame-by-frame basis, the process may not capturethese types of misalignments.

Differential resolution also may arise due to one or more of a varietyof reasons. For instance, the nature of the light path and lens coatingsin the Technicolor cameras typically caused the three film separationsto have drastically different resolution or sharpness. In particular,the cyan separation typically was located behind the yellow separationin what was known as a bipack arrangement. Light that passed through theyellow separation was filtered and unfortunately diffused beforestriking the cyan separation. As a result, the yellow (inverted blue)separation typically had a greater resolution compared to the cyan(inverted red) separation, and the magenta (inverted green) separationtypically had a resolution that was similar to that of the yellow(inverted blue) separation. This difference in resolution may result inred fringing that encircles many objects.

Loss of resolution may arise, for example, from the use of an opticalprinter. Such a loss of resolution will affect all three separations.Thus, the resulting printer output can never be as sharp as the source.

Digital image registration can be used to address one or more of thesefilm distortions. One aspect of digital image registration includes theprocess of aligning two or more digital images by applying a particularmapping between the images. Each digital image consists of an array ofpixels having a dimensionality that may be quantified by multiplying theimage width by the image height. Within the array, each pixel location(x, y), 0<=x<=width, 0<=y<=height, has an associated gray-level valueI(x, y), where 0<=I(x, y)<=65,535 (in the case of 16-bit data). Thegray-level value I(x, y) represents how much of the particular color(for example, red, green, or blue) is present at the corresponding pixellocation (x, y). If I1 represents a first image, I2 represents a secondimage, I1(x, y) represents a pixel value at location (x, y) within imageI1, and I2(x, y) represent a pixel value at location (x, y) within imageI2, the mapping between the two images can be expressed as:I2(x, y) corresponds to I1(f(x, y)),

where f is a two dimensional spatial coordinate transformation that canbe characterized by a pixel alignment vector. A registration algorithmmay be used to find a spatial transformation or alignment vector tomatch the images.

FIG. 1 illustrates one visual manifestation of misregistration that canoccur due to one or more sources of distortion. One implementation forperforming the registration in a way that reduces the misregistrationleading to such distortion is illustrated in FIG. 2, which illustrates asystem 200 including a digitization unit 210 that receives threeseparation images and outputs three digital, and possibly transformed,color component images. A feature selection unit 220 receives thesedigital images and, after processing, outputs them to an alignmentvector determination unit 230. The alignment vector determination unit230 determines transformations for two of the images against the third,with the third being used as a reference. In other implementations thatemploy other than three images, the alignment vector determination unit230 would produce transformations for N−1 images, where N is the totalnumber of images.

An alignment vector application unit 240 receives the twotransformations from the alignment vector determination unit 230 and thetwo non-reference digital images from the digitization unit 210. Thealignment vector application unit 240 modifies these two non-referenceimages using the transformations. Finally, a composite phase unit 250combines the two modified images and the reference image into acomposite image.

Digitization Unit

Multiple color separations are input into the digitization unit 210. Inone implementation, the digitization unit 210 accepts multiplephotographic negative images (for example, yellow, cyan, and magenta)and outputs multiple photographic positive images (for example, blue,red, and green) as digital data in the form of a set of gray-levelpixels. Other implementations may perform one or more of a variety ofother transformations, such as, for example, positive-to-negative, inlieu of or in addition to the negative-to-positive transformation;perform no transformation at all; or accept digitized data and therebyobviate the need for digitization.

Feature Selection Unit

Each of the digital color component images is input into the featureselection unit 220. Generally, the feature selection unit 220 selects afeature or feature set, such as, for example, one or more edges,objects, landmarks, locations, pixel intensities, or contours. In oneimplementation of the system 200, the feature selection unit 220identifies a set of edges, optionally or selectively refines this set,and outputs an edge map (labeled E_(R), E_(G), or E_(B)) that may be inthe form of an image consisting of edge and non-edge type pixels foreach color component image.

An edge detection filter, for example, a Canny filter, may beincorporated in or accessed by the feature selection unit 220 and may beapplied to a digital color component image in order to obtain a set ofedges. The edge information may be combined or separated into, forexample, orthogonal sets. One implementation obtains separate horizontaland vertical edge information.

After the edge detection filter has been applied and edge maps for theseparate color component images are created, the edge maps can befurther refined to attempt to identify a set of useful edges. Inparticular, the set of edges may be pruned to a smaller set so as toreduce the inclusion of edge pixels having properties that could causemisleading misregistration results. In one implementation, the featureselection unit 220 performs this pruning by applying one or morecriteria of merit to each edge in order to determine whether thatparticular edge should be included or rejected. Thereafter, one or moresecond criteria of merit may be applied to the collection of includededges in order to determine whether the entire set should be retained orif the entire set should be rejected. If there are no acceptable refinededges after using one or more refinement techniques either individuallyor in combination, the alignment vector can be determined in some othermanner, such as, for example, by applying the techniques discussed belowwith respect to the alignment vector determination unit 230.

Several techniques may be used to refine a set of edges by enforcing aminimum edge requirement and/or emphasizing high intensity areas.Examples of these techniques include the use of horizontal/verticalinformation and the use of high intensity selection, both of which arediscussed below.

a. Horizontal/Vertical Information

When searching for horizontal and vertical translational shifts, or moregenerally, alignments vectors, one implementation ensures that there isenough useful vertical and horizontal edge information within the areaunder consideration to make a useful alignment determination. Forexample, if there were only horizontal edges in an area (where the areacould be of any size up to the full image size), it may not bebeneficial to use these edges as features to determine a translationalshift in the horizontal direction.

In order to determine the usefulness of certain edges, each edge isfirst compared to a criterion of merit that determines the vertical andhorizontal extent of the edge in both absolute and relative (withrespect to the other direction) terms. Thereafter, the set of edges thathas been determined to have sufficient vertical or horizontal extent iscompared to another criterion of merit in order to determine whetherthis new set should be retained or rejected in its entirety.

For instance, in one implementation, determining the sufficiency ofvertical/horizontal edge information may include identifying a connectededge. For example, a connected edge may be identified by identifying aset of adjacent pixels that each have characteristics of an edge andthat each have at least one neighbor pixel with characteristics of anedge.

For each connected edge in the area under consideration, a determinationmay be made as to whether there is sufficient vertical and horizontalinformation. This determination may be made using the parameters that“min_x” is the minimum value of x within the connected edge,“y_for_min_x” is the value of y corresponding to the minimum value of x,“max_x” is the maximum value of x within the connected edge,“y_for_max_x” is the value of y corresponding to the maximum value of x,

“min_y” is the minimum value of y within the connected edge,“x_for_min_y” is the value of x corresponding to the minimum value of y,“max_y” is the maximum value of y within the connected edge,“x_for_max_y” is the value of x corresponding to the maximum value of y,“N_x” is max_x-min_x+1,“N_y” is max_y-min_y+1,“y_info” is the absolutevalue of (max_y−min_y)/(x_for_max_y−x_for_min_y), “x_info” is theabsolute value of (max_x-min_x)/(y_for_max_x−y_for_min_x), and T_x1,T_x2, T_y1, and T_y2 represent preset or configurable thresholds.

With these parameters, Total x, a value for the total number ofhorizontal edge candidate pixels, may be computed by adding N_x toTotal_x for each edge for which N_x is greater than T_xl and x_info_isgreater than T_x2. That is, an edge is included as a horizontal edgecandidate and the total number of horizontal edge candidate pixels isincremented by N_x if N_x and x_info_are greater than the thresholdsT_x1 and T_x2, respectively. Otherwise, none of the pixels for theconnected edge are used to determine vertical shifts.

Similarly, Total y, a value for the total number of vertical edgecandidate pixels, may be computed by adding N_y to Total_y for each edgefor which N_y is greater than T_y1 and y_info_is greater than T_y2. Thatis, an edge is included as a vertical edge candidate and the totalnumber of vertical edge candidate pixels is incremented by N_y if N_yand y_info_are greater than the thresholds T_y1 and T_y2, respectively.Otherwise, none of the pixels for the connected edge are used todetermine horizontal shifts.

Once all the edges of the area are processed, the total number ofcandidate edges for each direction, Total_x and Total_y, are compared tothe preset threshold, T_total.

If Total_x is greater than T_total, all of the pixels associated withthe identified horizontal edge candidates are considered horizontal edgepixels. Otherwise, if Total_x is less than or equal to T_total, thenumber of horizontal edge candidates is deemed insufficient, and, assuch, none of the edges within the area are used for the vertical shiftdetermination.

If Total_y is greater than T_total, all the pixels associated with theidentified vertical edge candidates are considered vertical edge pixels.Otherwise, if Total_y is less than or equal to T total, the number ofvertical edge candidates is deemed insufficient, and, as such, none ofthe edges within the area are used for the horizontal shiftdetermination.

Where no acceptable horizontal and/or vertical edges are identifiedwithin an area, an alternate method of obtaining the alignment valuesfor that area in one or more directions may be used. Several alternativemethods are discussed below with respect to the alignment vectordetermination unit 230.

b. High Intensity Selection

In general, a misregistration at bright areas of an image is moreobservable and objectionable than a misregistration at darker areas ofan image. For example, the eye would more readily observe a red areaextending beyond a white area than a red area extending beyond a brownarea. As such, it may be desirable to target or exclusively select edgesthat exist within high intensity areas. Such targeting/selection may beachieved through the construction of an edge map using a process thatcompares the gray-level pixel intensities associated with each colorcomponent image to a threshold. Although described below as beingapplied to an initial and thus unrefined edge map, high intensityselection may be applied to a refined edge map generated using thepreviously-described or some other refinement technique, individually orin combination.

For instance, in one implementation, RE_x indicates a particular pixelin the new refined edge map for the x color component image, where x canbe either red, green, or blue, E_x indicates a corresponding pixel inthe original edge map for the x color component image (where E_xcontains either edge or non-edge valued pixels), P_r indicates theoriginal gray-level intensity value for the corresponding pixel for thered component image, P_g indicates the original gray-level intensityvalue for the corresponding pixel for the green component image, P_bindicates the original gray-level intensity value for the correspondingpixel for the blue component image, and T_h indicates a preset pixelintensity threshold. In this implementation, RE_x is an edge pixel ifE_x is an edge pixel, P_r>T_h, P_g>T_h, and P_b>T_h. Otherwise, RE_x isnot an edge pixel.

However, because there may be misregistration, some edges that would becategorized as high intensity edges after correct alignment may not becategorized as high intensity edges before correct alignment. To avoidthis or other miscategorizations, the definition of a high intensityedge may be relaxed or expanded to be more inclusive. For instance, inone implementation, edge pixels within a window (of relatively smallhorizontal and/or vertical extent) relative to a high intensity edgealso may be categorized as high intensity edge pixels.

After the application of this process on each color component image, therefinement procedure for assessing horizontal and vertical edgeinformation can be applied to generate a more useful set of highintensity edges. Where there are not a sufficient number of useful highintensity edges within an area, the initial edge maps (that is, the edgemap obtained before the high intensity edge refinement process wasapplied) can be used instead. The edge refinement technique forassessing horizontal and vertical edge information then can be appliedto this edge map to obtain a useful set of edges within this area. Ifthere is not a sufficient number of edges in this case, an alternatemethod of obtaining the horizontal and/or vertical alignment for thatarea may be used, as discussed below. At the conclusion of edgerefinement, a new corresponding image consisting of edge and non-edgevalued pixels is created and transferred from the feature selection unit220 to the alignment vector determination unit 230.

Alignment Vector Determination Unit

The alignment vector determination unit 230 may operate on differenttypes of feature maps. Nonetheless, consistent with the examples setforth previously, operation of the alignment vector determination unit230 will be described in detail primarily for edges.

After the edges are obtained for each color component image, they arecompared between pairs of color component images in order to determinethe alignment vector that will lead that pair of images to be aligned,typically in an optimal manner. Other implementations may, for example,accept an alignment vector that satisfies a particular performancethreshold. In one implementation, each pair consists of one colorcomponent image that serves as a reference image, and a second colorcomponent image that serves as a non-reference image.

In the system 200, one color component image is maintained as areference image that does not undergo any alignment throughout the filmsequence, thus ensuring a constant temporal reference throughout thefilm sequence to be registered. The green reference image typically ischosen as the reference image due to its relatively high contrast andresolution. However, a red, a blue, or some other color component imagemay be selected as a reference, or the reference may be varied withtime. Other implementations may select a reference, if any, as warrantedby a particular application.

There are various possible spatial transformations that can be used toalign the color component images of a film frame. These include, forexample, affine (which includes, for example, translational, rotational,and scaling transformations), polynomial, or any part or combination ofthese types of transformations. In one implementation, thetransformation is represented as one or more translational alignments inthe horizontal and/or vertical directions. The transformation in thatimplementation can be described using I(x, y) to denote a pixelintensity at location (x, y) for a particular color component image, andlet I′(x, y) to denote a pixel intensity at location (x, y) after thetranslational alignment has been imposed on the color component image.With this notation, after the application of a translational alignmentvector of (deltax, deltay), I′(x+deltax, y+deltay) equals I(x, y), wheredeltax represents the horizontal alignment (displacement) and deltayrepresents the vertical alignment (displacement).

A translational transformation can be performed, for example, eitherglobally for the entire image or locally within different areas of theimage. In some instances relating to misalignment problems within film,the misalignment experienced at the outer areas of the image may differfrom the misalignment experienced at the center portion of the image. Assuch, in one implementation, different alignment vectors are applied todifferent areas of the image. In particular, localized alignment vectorsare determined for various areas of the image, as described below. Notethat a global alignment generally is a special case of the moregeneralized procedure that allows for local alignments.

In one implementation, the color component image is divided into areasarranged in a manner such that the center of at least one area and thecenter of at least one other area are in different proximity to thecenter of the image.

For simplicity of description, the case where areas are obtained bysegmenting the image into uniformly sized areas is considered below, butother segmentations or partitions also may be used. These areas can haveoverlapping pixels, that is, some pixels can belong to more than onearea within the non-reference image. Further, all areas of an image neednot necessarily be processed. Hereafter, the different areas of theimage will be referred to as blocks. FIG. 3 provides an example of animage 300 that is divided into sixty-four non-overlapping blocks.

In the alignment vector determination unit 230, for each block withinthe non-reference edge map image, a distortion value (or alternatively,a similarity value) is computed between a defined set of pixelsassociated with that block and the corresponding set of pixels in thereference image for a given translational alignment vector (deltax,deltay) using a registration metric such as that defined below. A pixelat location (x+deltax, y+deltay) in the reference image is defined to bethe corresponding pixel to a pixel at location (x, y) within a block inthe non-reference image for a translational alignment vector of (deltax,deltay). The set of pixels used to compute the registration metric valueassociated with the block can be a subset of the total pixels associatedwith the block.

One or more of various registration metrics (distortion/similaritymeasures) can be used. One general class of measures includesfeature-based measures that weight comparisons involving feature pixelsin a base image (reference or non-reference) differently, for example,more heavily, than comparisons involving non-feature pixels, where afeature pixel is a pixel determined to possess particularcharacteristics. One implementation uses a one-sided mismatchaccumulator as the distortion measure in order to ensure that themaximum potential distortion is constant for each tested (deltax,deltay) vector. In this implementation, the measure accumulatesdistortion for each pixel that satisfies the condition that there is apixel in the non-reference image that is identified as part of afeature, whereas the corresponding pixel in the reference image is notidentified as part of a feature. (Note that the term “part,” as well asany other similar term, is used in this application broadly to refer toeither “all” or “less than all.” For example, the above pixels may, ingeneral, contain all of the feature or less than all of the feature.) Inthis case, the maximum potential distortion for each tested vector wouldbe equal to the number of feature (e.g., edge) pixels within thenon-reference image.

One specific implementation is now described. Given a set of pixelsassociated with the non-reference image to be used in the distortioncalculation, for each pixel in this set, a determination is made as towhether the non-reference image contains an edge-type pixel when thereference image does not contain an edge-type pixel at the selectedcorresponding location for each image. If this case occurs, a positiveamount of distortion is assigned to this pixel. Otherwise, a distortionof zero is assigned to the pixel. The positive amount typically isconstant for all pixels in the set but may not be constant if, forexample, certain areas are to be emphasized or de-emphasized. Thedistortions for all pixels in the set are summed to obtain the totaldistortion value for a particular alignment vector.

Using the technique noted above, a total distortion value is computedfor a number of candidate (deltax, deltay) alignment vectors, within aparticular “window,” W, of size (2Wx+1)*(2Wy+1), where Wx, Wy areintegers greater than or equal to zero, the absolute value of Wx isgreater than or equal to deltax, and the absolute value of Wy is greaterthan or equal to deltay. The (deltax, deltay) vector that provides thelowest distortion value among the set of distortion values associatedwith the candidate alignment vectors is then selected as the alignmentvector, (deltax selected, deltay selected).

Given the one-sided mismatch accumulator distortion measure andassociated selection process, the alignment vectors in the associatedimplementation can be determined by determining an initial alignment,defined as (deltax_i, deltay_i), for the image. In one implementation,the center of the image is used to establish the initial alignmentvector upon which other blocks of the image base their alignment vector.As an example, the center can comprise the inner 25% of the image, whichmay overlap, partially or completely, an arbitrary number of blocks. Inparticular, given this portion in the non-reference image, the (deltax,deltay) pair that is chosen is the pair that provides the lowestdistortion using the one-sided mismatch accumulator distortion measureamong a number of candidate (deltax, deltay) vectors. If the candidatepairs are located within a window of 2*Wx_in+1 extent in the horizontaldirection and 2*Wy_in+1 extent in the vertical direction, then deltaxand deltay will satisfy:−Wx_in+deltax _(—) i<=deltax<=Wx_in+deltax _(—) i, and−Wy_in+deltay _(—) i<=deltay<=Wy_in+deltay _(—) i.

The alignment vectors for the individual blocks of the image aredetermined by processing the blocks in a radial manner. Because of thecolor fringing and larger alignment shifts that can occur toward theouter boundaries of the image in film implementations, the order inwhich the areas are processed and in which their alignments aredetermined is based, in one implementation, upon a radial path thatbegins near the center of the image and then progresses outward.Continuing the example given above, in which the non-reference image isdivided into sixty-four non-overlapping blocks, a radial ordering can beattained, for example, if the blocks are grouped into four differentrings. A radial ordering refers to processing blocks based on somemeasure of their distance from a chosen reference point, and, further,processing the blocks in either a generally increasing or a generallydecreasing distance from the chosen reference point, such as forexample, the center of an image. As discussed below, a radial orderingalso may process blocks randomly within a ring, where the rings areprocessed according to either a generally increasing or generallydecreasing distance using some measure of distance. An inner ring is aring that is positioned a smaller distance, using some measure, from thechosen reference point than a ring under consideration. Similarly, anouter ring is positioned a larger distance from a chosen reference pointthan a ring under consideration. An innermost ring has no ring that iscloser to the chosen reference point.

FIG. 4 illustrates four different rings. These rings are concentric. Thedetermination of the selected alignment vectors of the blocks proceeds,in this implementation, by first processing the blocks in the first ring(ring 0) consisting of blocks 27, 28, 35, and 36. Next, blocks withinthe second ring (ring 1) are processed, that is, blocks 18-21, 29, 37,45-42, 34, and 26. Subsequently, blocks within the third ring (ring 2)are processed, that is, blocks 9-14, 22, 30, 38, 46, 54-49, 41, 33, 25,and 17. Finally, blocks within the fourth ring (ring 3) are processed,that is, blocks 0-7, 15, 23, 31, 39, 47, 55, 63-56, 48, 40, 32, 24, 16,and 8. The manner in which each ring is processed may vary in differentimplementations. For illustrative purposes, a clockwise encirclement forthe different rings is demonstrated, that is, the blocks within aparticular ring are processed in a clockwise manner. For each block, atranslation alignment vector is determined by establishing an initialtranslation alignment vector for the block. In one implementation, theseinitial translation alignment vectors may be determined based on thealignment vectors of their neighboring block(s), where these neighborsbelong to the set of blocks that have already been processed and thatshare a common border or pixel(s) with the block under consideration.However, in other implementations, the blocks may not share a commonborder or pixel or the initial vector may be set by default or chosen atrandom.

The initial alignment vector for the block under consideration may beequal to a function of the neighbors of the block under considerationthat have already been processed. For example, if a clockwiseprogression is used, the set of neighbors for block 21 that have alreadybeen processed consists of blocks 20 and 28. Similarly, the set ofneighbors for block 6 that have already been processed consists ofblocks 5, 13, and 14. The function can be defined in a number of ways.For example, the function may be a weighting of the alignment vectorsamong each of the neighbors or the alignment vector of the neighbor thatprovides the minimum distortion for the block under consideration.

In implementations that emphasize the radial configuration, the neighborcan be chosen to be the inward radial neighbor of the current blockunder consideration. An inward radial neighbor is any neighboring blockhaving a distance, using some measure, that is no further from thechosen reference point than is the block under consideration. Thisimplies, for example, that the initial translational alignment vectorfor blocks 9, 10, and 17 would all be equal to the selectedtranslational alignment vector determined for block 18, and that theinitial translational alignment vector for block 11 would be equal tothe selected translational alignment vector determined for block 19.This may be computed, for example, by defining “numsideblocks” as aneven number representing the number of blocks in the horizontaldirection, denoting each block as “index” using an “index number”obtained by labeling the blocks within each image in a raster scanorder. For each outer ring m (m=1, 2, 3) and for each block n (n=0, . .. , 8*m+4) within ring m, where n increases in a clockwise direction asillustrated in FIG. 4, the neighbor for a particular block can bedefined as follows:

if (n is equal to 0) neighbor=index+numsideblocks+1;

else if (n<(2*m+1)) neighbor=index+numsideblocks;

else if (n is equal to(2*m+1)) neighbor=index+numsideblocks−1;

else if (n<(2*m+1)*2) neighbor=index−1;

else if (n is equal to(2*m+1)*2) neighbor=index−numsideblocks−1;

else if (n<(2*m+1)*3) neighbor=index−numsideblocks;

else if (n is equal to(2*m+1)*3) neighbor=index-numsideblocks+1; and

else neighbor=index+1.

For the inner ring, the initial estimate can be computed in a similarmanner or any other suitable manner. For example, the blocks within theinner ring can use the translational alignment vector determined for thecenter portion of the image as their initial estimate. In addition, thecenter portion of the image can use a preset initial alignment vector,which may be, for example, no initial displacement, or the displacementfor the central block or blocks of a previous frame.

Given the initial estimates, deltax_i and deltay_i and the associateddistortion for the block under consideration, the distortion associatedwith a number of candidate alignment vectors that represent differentdisplacements can be calculated. These alignment vectors are taken, forexample, from a set of vectors within a window described by thefollowing equations:−Wx(m, n)+deltax _(—) i<=deltax_selected<=Wx(m, n)+deltax _(—) i, and−Wy(m, n)+deltay _(—) i<=deltay_selected<=Wy(m, n)+deltay _(—) i.

In these equations, Wx and Wy are integers greater or equal to 0, andthe dependence of Wx and Wy on m and n indicates that the horizontal andvertical window areas can be different dimensions for different rings oreven different blocks within a ring. The alignment vector thatcorresponds to the displacement that produces the minimum distortionamong the candidate displacements chosen from this set then is selectedto be the alignment vector, (deltax_selected, deltay_selected), for theblock. Other implementations may use different selection criteria.

Recall that Wx_in and Wy_in represent the window sizes in the x and ydirections, respectively, that are used to determine the initialalignment vector for the entire image. If Wx(m)<Wx_in and Wy(m)<Wy_in,for m>=0, computational complexity is reduced because the set ofcandidate displacements is smaller in size. In one implementation, Wx(m,n) and Wy(m, n) for m, n>=0 are much less than Wx_in and Wy_in,respectively, resulting in a large increase in efficiency. In addition,by setting Wx(m, n) and Wy(m, n) to small values, the opportunity forvisible discontinuities between adjacent blocks may be decreased.

There are a number of strategies that may be employed to determine theselected candidate within a particular window of dimension(2*Wx+1)*(2*Wy+1). A straightforward approach is to check everydisplacement possibility within the window. Another implementation usesa spiral search with a partial distortion measure to determine theselected displacement or alignment vector. In particular, the differentdisplacements are considered in an order that begins at the locationassociated with the initial alignment vector and proceeds radiallyoutward in a spiral scanning path. Because the one-sided mismatchaccumulator is a cumulative distortion, it is possible to periodicallycompare the current minimum distortion to the distortion accumulatedafter only a partial number of the pixels within the block (that havebeen chosen to be used in the distortion calculation) have beenprocessed. If the partial distortion sum is found to be greater thanand/or equal to the current minimum distortion, then the candidatelocation cannot provide the minimum distortion and the other pixels inthe block need not be processed.

A spiral search with a partial distortion measure reduces thecomputational complexity associated with the search of all the candidatelocations. In particular, because the initial alignment vector is afunction of the neighboring blocks' selected alignment vectors, it islikely that the block under consideration will have lower distortionwith this alignment vector or with an alignment vector that correspondsto a displacement that is close in distance to this initial alignmentvector rather than an alignment vector that corresponds to adisplacement that is farther away in distance from the initial alignmentvector. As such, in the calculation of the distortion for a particularcandidate displacement, it is likely that the distortion will exceed thecurrent minimum value before a complete check of all of the pixelsassociated with the block that are chosen to be used in the distortioncalculation.

In another implementation, a method that does not search alldisplacement possibilities within the window can be used in order toreduce the computational complexity of the search. In oneimplementation, an iterative algorithm can be employed in which theselected alignment vector (corresponding to a particular candidatedisplacement) is first chosen for one direction (e.g., vertical), andthis result is then used as the initial alignment vector in the searchfor the selected alignment in the orthogonal direction (e.g.,horizontal), and this process is then iterated until a particularstopping condition is met. Such an implementation may use differentfeatures, for example, vertical and horizontal edges, to determine thealignment vectors for the horizontal and vertical directions,respectively.

For the case where separate horizontal and vertical edge information isretained, the following provides an example of a method that can be usedto select one candidate alignment vector from a set of candidatealignment vectors for a given block. First, initial conditions are set(step 1). In particular, the distortion_y associated with the initialalignment vector (deltax_i, deltay_i) is determined using the horizontaledge information, and minimum_y, the minimum distortion in the verticaldirection, is set equal to distortion_y, and the selected verticaldisplacement deltay selected(0) is set equal to deltay_i. In addition,deltax_selected(0) is set equal to deltax_i.

Then, for i=1, until the appropriate stopping condition is met, theselected vertical shift is determined using the horizontal edgeinformation (step 2-1). This may be done by calculating the distortion_yassociated with each of the candidate displacement possibilities(deltax_selected(i−1), deltay+deltay selected(i−1)) taken from the set−Wy<=deltay<=Wy, using the one-sided mismatch accumulator distortionmeasure (step 2-1-1). For the first iteration, it is not necessary tocalculate the distortion value, distortion_y, for deltay=0 because ithas already been calculated. For all other iterations, it is necessaryto calculate the distortion value associated with deltay=0.

The minimum distortion_y among the set of calculated distortion valuesthen is found, and deltay_selected is set to be the sum of the deltaythat produces this minimum distortion value and deltay_selected(i−1),and this distortion value is set to be the new minimum distortion value,minimum_y (step 2-1-2). A determination then is made as to whether thestopping condition for a particular direction has been met (step 2-1-3).If so, step (2-1) will not be repeated after step (2-2).

Next, the selected horizontal shift is determined using the verticaledge information (step 2-2). This is done by calculating distortion_xassociated with each of the candidate displacement possibilities(deltax+deltax selected(i−1),deltay selected(i−1)) taken from theset−Wx<=deltax<=Wx, using the one-sided mismatch accumulator distortionmeasure (step 2-2-1).

The minimum distortion_x among the set of calculated distortion valuesthen is found, deltax_selected is set to be the sum of the deltax thatproduces this minimum distortion value and deltax_selected(i−1), andthis distortion value is set to be the new minimum distortion value,minimum_x (step 2-2-2). A determination then is made as to whether thestopping condition for a particular direction has been met. If so, step(2-2) will not be repeated after step (2-1).

Each stopping condition can be, for example, based on a preset number ofiterations or on the condition that the selected alignment vector for ablock for iteration i is the same as that for the preceding iteration,for example, when deltax_selected(i−1)=deltax_selected(i) anddeltay_selected(i−1)=deltay_selected(i), among others.

If there is not a similar implementation that reduces the number ofcandidate locations searched can be performed using edge informationthat is captured for both vertical and horizontal directionssimultaneously (for example, the edge information is based on themagnitude of the edge strength). In such a case, the distortion valuecomputation at (deltax_selected(i−1), deltay_selected(i−1)) foriteration i need not be calculated because it already has beencalculated in iteration i−1, and the single edge-map information is usedinstead of the horizontal and vertical edge maps discussed above. Ifthere is not a sufficient number of useful edges in a particulardirection within a block to be registered within the non-reference imageor within a corresponding block in the reference image, an alternativemethod may be performed to select an alignment vector for this block.For example, the alignment for that direction can simply be taken to bethe initial alignment for that direction. Alternatively, a larger areaencompassing more blocks (or even the entire image) can be used todetermine the selected alignment vector for this block. Otheralternative or additional methods can also be used to select analignment vector for this block. In one implementation, if the centerportion of the image does not have enough useful edges, the alignmentvector for the center is set to the selected alignment vector determinedfrom the entire image.

Alignment Vector Application Unit

After the alignment vector determination unit 230 determines thealignment vector for each block within the color component image, thealignment vector application unit 240 aligns each block using thesevectors or a modification of them. If, for example, only one blockexists within the image and only one global alignment vector iscomputed, then the alignment is straightforward. If, however, the imagehas been segregated into multiple blocks, the image can be spatiallyaligned by a number of different methods.

One technique that can be used for alignment involves applying a uniformalignment to each block by its corresponding alignment vector. However,different blocks within that color component image may have differentalignment vectors. In such a case, discontinuities may exist at aboundary between blocks.

The alignment vector application unit 240 attempts to reduce theperceptual effects of discontinuities by “feathering” at the boundariesbetween blocks with different alignment vectors, as described below. Inthe implementation of system 200, this technique is applied only to thenon-reference images because the reference image does not undergo anyalignments (recall that the non-reference images are shifted withrespect to the reference image).

Although the feathering process will be described hereinafter withreference to horizontal and vertical alignment values to maintainconsistency with early examples, the feathering process also isapplicable to other types of transformations. Feathering is performed,in this implementation, along the y-axis boundaries between twohorizontally neighboring blocks, along the x-axis boundaries between twovertically neighboring blocks, and for the four-corner boundariesbetween four neighboring blocks.

For instance, FIG. 5 provides an example 500 identifying several pixelsthat are affected by the feathering scheme described above when theimage is divided into sixteen uniformly sized areas. For each y-axisboundary, feathering is performed across a particular-sized horizontalwindow. For example, for the y-axis boundary between blocks 5 and 6, thewindow 510 may be used. For each x-axis boundary, feathering isperformed across a particular-sized vertical window. For example, forthe x-axis boundary between blocks 2 and 6, the window 520 may be used.For each of the four-corner boundaries, feathering is performed across aparticular-sized vertical and a particular-sized horizontal window. Forexample, for the four-corner boundary between blocks 10, 11, 14, and 15,the window 530 (arrows point to corners of window 530 in FIG. 5) may beused. The sizes of these “feather windows” typically impact the rate atwhich the alignment values for the pixels at the boundaries blend fromone value to another. In one implementation, the window size isdetermined as a function of the maximum (max) of the difference betweenthe x alignment values of the neighboring blocks and the differencebetween the y alignment values of the neighboring blocks. However, manytechniques may be used to determine the size and/or shape of the variouswindows. These windows need not be rectangular or continuous.

In one implementation, within the feather window, new alignment valuesare obtained by linearly interpolating the different alignment values ofthe neighboring blocks under consideration. Another implementation usesnon-linear interpolation. In either case, the interpolated alignmentvalues then are used to obtain the new intensity value of the pixel at aparticular location. In particular, the pixel at the locationcorresponding to the selected alignment value is used as the value forthe current location. If the selected alignment value in a particulardirection is not an integer, then the intensity values of the pixelsthat correspond to the two integer-valued displacements closest indistance to the selected displacement are appropriately weighted andcombined to obtain the final new intensity value.

Note that the calculation of the new alignment vectors within thefeathering window may be performed in the alignment vector determinationunit 230 rather than the alignment vector application unit 240. Thecalculation of the new intensity values within the feathering windowwould still be performed in the alignment vector application unit 240.Other implementations may perform many of the various describedoperations in different orders or in different functional blocks.

As an example of the feathering scheme, assume (dx1, dy1) is thealignment vector for block 5, and (dx2, dy2) is the alignment vector forblock 6. In this example, feathering across the y-axis boundary that isshared between these two blocks is addressed, and for simplicity, thecorner conditions are not addressed. Then the size of the feather windowcan be “fwsize”=constant*max(abs(dx1−dx2), abs(dy1−dy2)), where abs( )indicates the absolute value function. Assume the boundary location isat (x3, y3) and the height of the block is “blockh.” Then, the alignmentvalues will be interpolated from the x3−(fwsize/2) position to thex3+(fwsize/2) position in the x direction for each of the rows from thevertical start of the block to the vertical start of the block+blockh.For example, assume that max(abs(dx1−dx2), abs(dy1−dy2))=abs(dx1−dx2).Then, the horizontal alignment value for the point (x4, y4) is computedas (dx1+(x4−(x3−(fwsize/2)))*(dx2−dx1)/(fwsize)) and the verticalalignment value for the point (x4, y4) is computed as(dy1+(x4−(x3−(fwsize/2)))*(dy2−dy1)/(fwsize)). The new value at aparticular location is the value of the pixel at the calculateddisplacement location (or the weighted combination of the intensities atthe nearest integer grid points). Note that special care may need to beapplied to the boundaries of the image when the feathering approach isused.

Another implementation adjusts for discontinuities by using warping.Although the warping process will be described hereinafter withreference to horizontal and vertical alignment values to maintainconsistency with early examples, the warping process also is applicableto other types of transformations. In one example of a warpingtechnique, each block can be identified with a control point at itscenter. The horizontal and vertical alignment values that were obtainedfor each block can become the alignment values for the block's controlpoint. The alignment values for the remaining pixels within the imagemay be obtained by interpolating the alignment values of the nearestcontrol points. These alignment values are then applied to the pixelswithin the non-reference image.

Composite Phase Unit

Once the non-reference images are aligned by the alignment vectorapplication unit 240, the images can be recombined into a compositecolor frame by the composite phase unit 250. FIG. 6 illustrates acomposite frame after the composite phase unit 250 has been applied tothree color component images. In one implementation, a laser filmprinter is optionally used to avoid the loss of resolution incurred withan optical printer.

Additional Implementations

The implementations and techniques described above can be applied to avariety of applications in which multiple separations need to beregistered. Examples include spectral and non-spectral separations.Spectral separations are used, for example, in: (1) color filmapplications capturing, for example, different color frequencies, (2)astronomical applications capturing, for example, radio frequenciesand/or optical frequencies, and (3) medical applications capturing, forexample, different magnetic (MRI), X-ray, and sound (ultrasound)frequencies. As the last example illustrates, spectral separations maybe captured from various frequency sources, including, for example,electromagnetic and sound waves. Non-spectral separations may beobtained from, for example, variations in pressure, temperature, energy,or power.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the claims. For example, theimplementations and features described may be implemented in a process,a device, a combination of devices employing a process, or in a computerreadable medium or storage device (for example, a floppy disk, a harddisk, RAM, ROM, firmware, electromagnetic waves encoding or transmittinginstructions, or some combination) embodying instructions for such aprocess.

One such device is, for example, a computer including a programmabledevice (for example, a processor, programmable logic device, applicationspecific integrated circuit, controller chip, ROM, or RAM) withappropriate programmed instructions and, if needed, a storage device(for example, an external or internal hard disk, a floppy disk, a CD, aDVD, a cassette, a tape, ROM, or RAM). The computer may include, forexample, one or more general-purpose computers (for example, personalcomputers), one or more special-purpose computers (for example, devicesspecifically programmed to communicate with each other), or somecombination.

Accordingly, other implementations are within the scope of the followingclaims.

What is claimed is:
 1. A method of processing component images, themethod comprising: accessing a first component image, the firstcomponent image being a digital color component image of a scene andcorresponding to a first color; accessing a second component image thatis separate from the first component image, the second component imagebeing a digital color component image of the scene and corresponding toa second color that is different than the first color; accessing a thirdcomponent image that is separate from the first component image and thesecond component image, the third component image being a digital colorcomponent image of the scene and corresponding to a third color that isdifferent than the first color and the second color; transforming thefirst component image, the second component image, and the thirdcomponent image to generate a transformed version of the first componentimage, a transformed version of the second component image, and atransformed version of the third component image; aligning thetransformed version of the first component image, the transformedversion of the second component image, and the transformed version ofthe third component image; and based on the alignment of the transformedversion of the first component image, the transformed version of thesecond component image, and the transformed version of the thirdcomponent image, recombining, by at least one processor, the transformedversion of the first component image, the transformed version of thesecond component image, and the transformed version of the thirdcomponent image into a composite color image of the scene.
 2. The methodof claim 1, wherein the first component image corresponds to a redcolor, the second component image corresponds to a blue color, and thethird component image corresponds to a green color.
 3. The method ofclaim 1, further comprising: digitizing the first component image;digitizing the second component image; and digitizing the thirdcomponent image.
 4. The method of claim 1, wherein aligning thetransformed version of the first component image, the transformedversion of the second component image, and the transformed version ofthe third component image comprises determining automatically analignment vector for at least a part of a selected component image fromamong the accessed first, second, and third component images.
 5. Themethod of claim 4, wherein the alignment vector aligns the part of theselected component image with a corresponding part of a second of theaccessed first, second, and third component images, the method furthercomprising combining the selected component image and the second of theaccessed first, second, and third component images after applying thealignment vector.
 6. The method of claim 4, further comprising selectingone of the accessed first, second, and third component images as areference, wherein determining the alignment vector comprisesdetermining an alignment vector between a part of the reference and thepart of the selected component image.
 7. The method of claim 1, furthercomprising storing the composite color image of the scene on amachine-readable medium.
 8. An apparatus for processing componentimages, the apparatus comprising at least one processor programmed toperform operations comprising: accessing a first component image, thefirst component image being a digital color component image of a sceneand corresponding to a first color; accessing a second component imagethat is separate from the first component image, the second componentimage being a digital color component image of the scene andcorresponding to a second color that is different than the first color;accessing a third component image that is separate from the firstcomponent image and the second component image, the third componentimage being a digital color component image of the scene andcorresponding to a third color that is different than the first colorand the second color; transforming the first component image, the secondcomponent image, and the third component image to generate a transformedversion of the first component image, a transformed version of thesecond component image, and a transformed version of the third componentimage; aligning the transformed version of the first component image,the transformed version of the second component image, and thetransformed version of the third component image; and based on thealignment of the transformed version of the first component image, thetransformed version of the second component image, and the transformedversion of the third component image, recombining the transformedversion of the first component image, the transformed version of thesecond component image, and the transformed version of the thirdcomponent image into a composite color image of the scene.
 9. Theapparatus of claim 8, wherein the first component image corresponds to ared color, the second component image corresponds to a blue color, andthe third component image corresponds to a green color.
 10. Theapparatus of claim 8, wherein the operations further comprise:digitizing the first component image; digitizing the second componentimage; and digitizing the third component image.
 11. The apparatus ofclaim 8, wherein aligning the transformed version of the first componentimage, the transformed version of the second component image, and thetransformed version of the third component image comprises determiningautomatically an alignment vector for at least a part of a selectedcomponent image from among the accessed first, second, and thirdcomponent images.
 12. The apparatus of claim 11, wherein the alignmentvector aligns the part of the selected component image with acorresponding part of a second of the accessed first, second, and thirdcomponent images, the operations further comprise combining the selectedcomponent image and the second of the accessed first, second, and thirdcomponent images after applying the alignment vector.
 13. The apparatusof claim 11, wherein the operations further comprise selecting one ofthe accessed first, second, and third component images as a reference,wherein determining the alignment vector comprises determining analignment vector between a part of the reference and the part of theselected component image.
 14. The apparatus of claim 8, wherein theoperations further comprise storing the composite color image of thescene on a machine-readable medium.
 15. A non-transitorycomputer-readable medium encoded with a computer program comprisinginstructions that, when executed, operate to cause a computing device toperform operations comprising: accessing a first component image, thefirst component image being a digital color component image of a sceneand corresponding to a first color; accessing a second component imagethat is separate from the first component image, the second componentimage being a digital color component image of the scene andcorresponding to a second color that is different than the first color;accessing a third component image that is separate from the firstcomponent image and the second component image, the third componentimage being a digital color component image of the scene andcorresponding to a third color that is different than the first colorand the second color; transforming the first component image, the secondcomponent image, and the third component image to generate a transformedversion of the first component image, a transformed version of thesecond component image, and a transformed version of the third componentimage; aligning the transformed version of the first component image,the transformed version of the second component image, and thetransformed version of the third component image; and based on thealignment of the transformed version of the first component image, thetransformed version of the second component image, and the transformedversion of the third component image, recombining the transformedversion of the first component image, the transformed version of thesecond component image, and the transformed version of the thirdcomponent image into a composite color image of the scene.
 16. Thecomputer-readable medium of claim 15, wherein the first component imagecorresponds to a red color, the second component image corresponds to ablue color, and the third component image corresponds to a green color.17. The computer-readable medium of claim 15, wherein the operationsfurther comprise: digitizing the first component image; digitizing thesecond component image; and digitizing the third component image. 18.The computer-readable medium of claim 15, wherein aligning thetransformed version of the first component image, the transformedversion of the second component image, and the transformed version ofthe third component image comprises determining automatically analignment vector for at least a part of a selected component image fromamong the accessed first, second, and third component images.
 19. Thecomputer-readable medium of claim 18, wherein the alignment vectoraligns the part of the selected component image with a correspondingpart of a second of the accessed first, second, and third componentimages, the operations further comprise combining the selected componentimage and the second of the accessed first, second, and third componentimages after applying the alignment vector.
 20. The computer-readablemedium of claim 18, wherein the operations further comprise selectingone of the accessed first, second, and third component images as areference, wherein determining the alignment vector comprisesdetermining an alignment vector between a part of the reference and thepart of the selected component image.